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(57) ABSTRACT 

A computer operable method for integrating and automating 
test procedures within a computer application program. 
Instantiated test operation objects of an object class denned 
by the present invention correspond to functions to be tested 
within the computer application program. The test operation 
objects are instantiated by calls to functions in a test opera- 
tion runtime library (DLL). The test operation objects 
include interface method functions which execute the 
intended test operation and simulate any required data, file, 
or memory I/O. Other aspects of the methods of the present 
invention provide rules which permit decisions as to the 
applicability of each test operation given the state of the 
application program or the context of the test operation 
sequence. The various test operation objects are selected to 
perform a sequence of test steps. In one mode of operation, 
the methods of the present invention randomly select among 
all the instantiated test operation objects. In another mode of 
operation, the methods of the present invention "playback" 
a previously recorded sequence of test operation objects to 
permit reproduction of failures in previous test sequences. In 
a third mode of operation, the methods of the present 
invention permit a user to modify or create "playback" files 
to customize a test case for a particular focus. 

3 Claims, 12 Drawing Sheets 
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METHOD FOR INTEGRATING AUTOMATED 
SOFTWARE TESTING WITH SOFTWARE 
DEVELOPMENT 

RELATED APPLICATIONS 5 

This application is a continuation of Rodrigues et al. U.S. 
Ser. No. 08/552,483, filed Nov. 9, 1995, which issued as 
U.S. Pat. No. 6,067,639 on May 23, 2000. 

FIELD OF THE INVENTION 10 

The present invention relates to the field of computer 
software development and testing, and in particular, to a 
method for integrating automated software testing into a 
product as it is being developed to thereby simplify subse- 15 
quent software product testing. 

PROBLEM 

It has been common in the computer software arts that 
software product development was a distinct and separate 2 o 
process from software product testing. In the standard soft- 
ware product development lifecycle, development engineers 
would iteratively develop and refine a software product 
according to requirement or functional specifications. The 
development engineers frequently would test the product 25 
under development either in an ad-hoc manner or in a more 
rigorous manner. In either case, when the product was 
deemed to be completed by the development group of 
engineers, the software product was turned over to the 
testing process, usually a separate set of engineers from the 30 
group that developed the software product. Most, if not all, 
of the testing process utilized in the development phase 
would be discarded and the test engineers would begin anew 
evaluating the software product as against its corresponding 
product specifications. 3S 

In a typical test environment, the test engineers are 
provided with little or no internal information regarding the 
structure and design of the code comprising the software 
product. The testing effort would then proceed in a "black 
box" manner by applying test data (an external stimulus) and 40 
observing the output of the software product for proper 
response. In its crudest form, a test engineer determines 
("dreams up") potentially interesting test inputs and applies 
them to the software product while observing the output 
behavior of the software product for proper operation. This 45 
form of manual testing permits wide flexibility for the test 
engineer in creating input stimuli but provides little assis- 
tance to the test engineer in reproducing problems found 
during the test sequence. Manual test requires the test 
engineer to carefully note the sequence of test inputs which 50 
led to a specific problem test result. 

Test engineers have developed or utilized a number of 
tools to assist in such "black box" testing. To automate the 
above discussed manual test process, scripting and macro 
languages are frequently used. The script/macro tool allows 55 
some degree of automation in the test sequence to aid in 
reproducing intermittent failures. A macro tool permits some 
added flexibility in the use of variables to customize the 
operation of the script based on externally supplied variable 
inputs. Some software products (e.g. word processing pro- 60 
grams or spreadsheet programs) offer built-in macro features 
to reproduce sequences of user input keystrokes. Other 
program user interface (UI) environments (such as the 
Xwindows programming environment) provide for redirec- 
tion of a program's input from a script/macro file and thus 65 
may automate the testing of specific user keystroke 
sequences. 
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2 

Simple "BAT* files in the MS-DOS® environment or 
"shell scripts" in the UNIX® environment are exemplary of 
such scripting tools used to partially automate the software 
product testing process. Macro facilities in the "BAT" file or 
"shell script" interpreters and other macro replacement 
programs such as "m4" or "perl" in the UNIX® environment 
are exemplary of macro facilities used to enhance the 
scripting facilities in automating software product testing. 

However, such scripting/macro based testing tools still 
provide little or no assistance to the test engineer in observ- 
ing the software product output to automatically detect 
success and failure of each test. Collection and analysis of 
the test results remains largely a manual procedure. Corre- 
lation of the gathered test output with the timing and 
operation of the software product is difficult if not impos- 
sible. 

An additional problem with scripting/macro tools arises in 
that the tool itself as well as the test scripts themselves must 
be ported to each computing environment in which the 
software product is to be tested. For example, a particular 
operation in a PC based Microsoft Windows® application 
may be tested by simulating an "OPEN" menu operation 
while a similar function may require an "OK" menu opera- 
tion in an Apple Macintosh® computing environment. The 
test scripts would have to change to test the same function 
in these two exemplary computing environments. In other 
words, the testing of underlying functions of an application 
may have to change as the user interface changes over a 
variety of computing environments. This creates an addi- 
tional burden in porting and maintaining the test tools and 
test scripts along with the software products. In addition, the 
porting and maintenance of the scripting/macro tools and 
test cases can be a source of additional errors which may be 
erroneously attributed to the failure of the software product 
under test. 

Scripting/macro based test tools also tend to vary depend- 
ing upon the external stimuli needs of each software product 
under test. The user interface for the test engineer using 
"black box" automated test tools tends to vary as the needs 
of each software product vary. Test engineers therefore must 
potentially learn a new user interface mode of operation for 
each software product to be tested. 

All such "black box" testing methods, with or without the 
aid of automated scripting/macro testing tools, are typically 
performed without knowledge of the software product's 
internal code structure. This lack of structural knowledge 
can make testing more cumbersome and time consuming. 
The lack of structural knowledge precludes certain styles of 
testing which may focus on particular error prone aspects of 
the software product revealed only by knowledge of the 
software product's internal structure. Sometimes, for 
example, an error in one operation of the software product 
may not be externally observable in the output of the 
software product until subsequent operations are performed. 
The combination of the operations may eventually reveal the 
problem in an external manifestation, but the sequence of 
event may be lengthy back to the actual cause of the 
problem. Thus the use of external stimuli ("black box") test 
methods, even in association with various automated test 
tools, does not permit the test engineer to exercise judge- 
ment with respect to problems more easily located through 
testing with knowledge of the software product's internal 
code structure. 

In addition, as software products evolve with ever increas- 
ing complexity, the burden of testing is being shifted more 
toward the development teams. It is simply impossible with 
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"black box" testing techniques to exhaustively test all pos- The application programmer determines the operations to 

sible inputs (external stimuli) of many software products be tested in the application program and instantiates a test 

regardless of the size of the testing staff. Even testing of a operation object (of a predefined test API object class) 

large portion of the possible inputs is a difficult task in many corresponding to each operation to be tested. The test 

cases. Therefore, the software development/testing lifecycle 5 operation object identifies an execute interface method 

has begun to shift more of the testing efforts onto the (provided by the application programmer) to be invoked to 

responsibility of the development staff. For example, it is execute the desired operation. Other interface methods of the 

more frequent now that "acceptance testing"(testiug for test operation object identify functions which are to be 

fundamental functionality) is made a part of the responsi- performed to simulate the generation of input data and the 

bility of the development staff. An added benefit of shifting 1Q acceptance of output data in response to operation of the 

some test burden to the development group is realized in the execute interface method. For example, one interface 

knowledge of internal code structure of the software prod- me thod provides functions to be invoked for the manual or 

uct.Thecodestnicmreknowledgeofthedevelopmentgroup automa tic generation of input data to test the operation, 

may be applied to focus the testing effort on test sequences AnQther interface method of ^ ^ ^ object 

and scenarios most likely to ^create problems. In addition the yidcs tQ be inyokcd m fe to I/Q b of 

developers may construct *e test cases to detect the failure * ^ Qn test Another inte rf ace method of the test 

in the test sequence as early as possible. It is a problem for \. ,. L , „ , r . A , . . 

developers to create a standardized test interface for the °P eratl ° n ob ^ ct defines a of rules to permit decisions to 

automated performance of test sequences in a manner that be > made m invocat i°n of the test ob J ect * 

permits easy reproduction of the failed test sequences. ^ application program is designed by the development 

It is apparent from the above discussion that a need exists 20 engineers with the test operation objects defined as part of 

for an automated testing tool which permits a standardized the application program. The application program is then 

interface for the generation and execution of test cases, compiled and linked with the test tool library (DLL) to 

which permits random sequences of testing, which permits provide the standard object class manipulation test methods 

easy reproduction of failed test sequences, and which auto- of the present invention. By defining the test operation 

matically senses the success or failure of each test step. 2 5 objects as discussed above, the application programmer has 

SOLUTION provided information (in the form of a collection of instan- 
tiated objects) useful to the test methods of the present 

The present invention solves the above identified prob- invention to invoke each test operation and detect success or 

lems and others to thereby advance the state of the useful failure of each tested operation> since the test operations are 

arts by providing an easy to use, coding standard and user de&i d ^ the application program and the library func . 

interface for a software developer to integrate testing of a tions which . lement thc test tion classes of thc 

software product into the development 01 the product. Ine , . . n , T • . r nf 

. 1 • > ■ . . r ■ fle t *r>T\ - present invention are available in a variety or computing 

testing application program interface (test API) is used by an r . A ±t A . ' 1 

application program developer to integrate a standardized environments, the application program may be more easily 

test interface into the software product in its development tested m many environments without the need to port unique 

phase. Use of the test API by the application program 35 test tools to each new computing environment. The test tools 

invokes standard functions in a dynamic linked library can be readll y available m any computing environment to 

(DLL) to initiate a standard test user interface when the wm ' cn tne development staff chooses to port the application 

application program is run. The test tools are integrated into program. In addition, the test tools of the present invention, 

the application program and are therefore ported to any new unlike scripting and macro methods of the past, do not 

computing environment by simply porting the application 40 require the development or testing teams to port the test 

program to that environment. The development engineer, cases between the various computing environments on 

possibly in cooperation with the test engineers, thereby which the program is to be tested. The test tools of the 

designs the test capability into the application program from present invention are available for all platforms on which the 

its inception. These test hooks, combined with the standard, application program may be run. The test suites are recorded 

easy to use, graphical user interface of the test tools of the 45 in a portable manner by the test tools of the present invention 

present invention, permit any engineer in the product devel- in so-called playback files. 

opment team to test the product. Development engineers ' The application program is designed by the application 
may more effectively test the application program during its programmer to invoke the test tool at an appropriate stage in 
development phase to reduce the number of problems dis- the runtime-startup or processing of the application pro- 
covered later in the product's lifecycle. Test engineers may 50 gram. The test tool may then be operated in one of three 
apply the same easy to use graphical interface to more modes: namely, random automated test operation selection, 
thoroughly test the functionality of the product. playback automated test operation selection, or manual 

The test API functions in the DLL provide a consistent, playback creation mode, 
friendly, easy to use interface for testing of the application In the random automated test selection mode, the test 
program. The DLL test functions may be operated in a 55 methods of the present invention pseudo randomly select 
random mode to generate pseudo-random test sequences. from among the set of instantiated test operation objects 
The DLL test functions automate recording of the pseudo- defined by the application programmer. As each test opera- 
random seed and the corresponding sequence of test steps tion object is randomly selected, the execute function 
for easier reproduction of problems generated in the random method associated with the object is invoked to perform the 
sequence. The DLL functions can be used to automatically 60 developer's designed test function. The random selections 
determine the success or failure of the corresponding test are logged in a playback file to permit later reproduction of 
step by sensing a return value from the invocation of a a failed sequence. In addition, the results of each execute 
corresponding sequence of execution in the application function method invocation are logged in a file by the 
program. In addition, the application program may directly methods of the present invention so that a sequence may be 
invoke other DLL functions to record an application failure. 65 reproduced if it produces a problem in the application 
This alternative method is referred to herein as the assertion program under test. This process continues randomly select- 
method, ing test operation objects for which the corresponding 
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execute function is invoked until directed to stop by a user assurance or a testing team on the right side of the dashed 

or other termination conditions are met. line. It is axiomatic that the development team has extensive 

In the playback automated test operation, the test opera- knowledge of the internal structure of the application pro- 

tion selections made in a previous test sequence are replayed gram. By contrast, it is common in present software devel- 

from the playback file created during an earlier performance 5 opment and test environments that the quality assurance or 

of the test sequence. This mode of operation is used, for testing team operates using "black box" techniques 

example, to permit reproduction of test sequences which led (applying stimuli and observing results without such knowl- 

to erroneous results in previous invocations. edge of the structure of the application program). In this 

The manual playback creation mode of operation permits { p ic * 1 software development and test environment, the 

a user of the present invention to create or edit a playback 10 development team helps determine the requirements and 

file. This mode allows the user finer control over the test functional specification of the application program as 

sequencing to more rapidly debug an application program. de P lct J ed ™ element 100 In element 10 ^] he development 

. , , team develops appropriate programs and data structures to 

The methods of the present invention enable automation taplement the requirem ents and functional specifications. In 

of the testmg of an application program while integrating the u conjuaction ^ the deve i opment thereo f, the development 

test cases with the program development. This integration of , eam verifles of valida(es the m ^ , he jre . 

the test case and program development permits more thor- meDts and specifications . ]n this typical applica- 

ough analysis and regression of the application program. ^ development process, the development team 

The automation aspects of the test methods of the present has , hus completed its task and provides me app ] ication 

invention permit easier, unattended test operation to more m ^ tQ ^ u assurance or testin team for inde . 

effectively test an application program. pendent testing and verification of the appU cation program. 

BRIEF DESCRIPTION OF THE DRAWINGS "? element 104 r ff* .f^T' or te ? tin S te f m 

performs a regression test to verify that previously reported 

FIG. 1 is a flowchart depicting the lifecycle flow of an and fixed problems have not recurred in this new version of 

application program through development and test as pres- 25 the application program. Next, at element 106, the quality 

ently known in the art; assurance or testing team performs new testing of the 

FIG. 2 is a flowchart depicting the lifecycle flow of an application program to reveal potential new problems aris- 

application program through development and test as ing in the current version of the application program. Due to 

improved by the tools and methods of the present invention; the nature of the prior testing tools, the test team must 

ttt^ a • ui i j- i • i* > i , 30 initiate measures at element 108 (often manual) to document 

Mu. 3 is a block diagram depicting the relationship 4 , . , „ - \, ' 

. ■ * j * 1 1 ± * i j v „• tne steps required to reproduce the new problems discov- 

between script oriented external test tools and an application ^ r n r r 

program as known in the art; ' . t 4 . 1% t 

At element 110, the quahty assurance or testing team 

FIG. 4 is a block diagram depicting the relationship determines whcther or not m6 number mA S6V6rity of 

between script oriented external test tools and an application 35 revea i e d problems are too high to permit release of the 

program as improved by the tools and methods of the present applicatioil program to potential users. If it is determined 

invention; t ^ at ^ e numDer anc j severity of problems revealed is not too 

FIG. 5 is a flowchart depicting the control flow of an high, the quality assurance or testing team will release the 

application program with test operation objects embedded appUcation program to potential users as depicted in element 

therein in accord with the present invention; 40 114. If the number or severity of the problems revealed is too 

FIG. 6 A is a flowchart showing additional detail of the high, the quality assurance or testing team does not release 

random test operation element of FIG. 5; the application program to potential users, but rather reports 

FIG. 6B is a flowchart showing additional detail of the the °P en problems to the development team as depicted in 

random test operation of FIG. 5; element 112. In parallel with release of the application 

FIG. 7 is a flowchart showing additional detail of the 45 f r ° t ? rara ; asdc P^ d » elcmcn < ^ the q^hty assurance or 

playback test operation of FIG. 5; f ^ team ^ ^° fP ort f °P en u P roblems X ? the 

L„ 0 . a . . . ... . developers as shown by element 112. With the report of open 

FIG. 8 is a flowchart showing additional detail of the problems returned to the development team, the develop- 

manual test operation of FIG. 5; mcnt team fcpcats {{& prcvicms efforts m element 102 to 

FIG. 9 is a screen dLsplay showing an exemplary user 50 re-design the application program to resolve problems 

interface which permits the user to define group and test reported by the quality assurance or testing team. This cycle, 

operation weights for the random test operations of FIG. 5; elements 102-112, repeats until the quality assurance or 

FIG. 10 is a screen display showing an exemplary user testing team finds no remaining problems in the application 

interface which permits the user to define options for the program worthy of reporting to the development team, 

random test operations of FIG. 5; and 55 FIG. 3 depicts a typical computing environment 308 in 

FIG. 11 is a screen display showing an exemplary user which the 1 ualit y assurance or testing team performs their 

interface which permits the user to customize a playback test testin g function. A typical testing approach involves the use 

sequence for the playback test operations of FIG. 5. of a ? cri P l oriented test tool 304 to apply external stimuli to 

application program 306. The application program receives 

DETAILED DESCRIPTION OF THE 60 the externally applied stimuli as a simulated user input and 

INVENTION generates an appropriate response by performing its desig- 

OVERVIEW — PRIOR ART: nated function on that provided input. The script oriented 

FIG. 1 describes the overall flow of a typical software test tool 304 detects the response generated by the applica- 

development and test life cycle for an application program tion program 306 to verify proper operation of the applica- 

product. A vertical dashed line in the middle of FIG. 1 65 tion program 306 in response to the applied stimuli. Script 

delineates tasks performed by a development team on the oriented test tool 304 may receive inputs from test suite or 

left side of the dashed line, and tasks performed by a quality script files 300, which describe a sequence of test steps to be 



05/21/2004, EAST Version: 1.4.1 



US 6,408,403 Bl 



8 



executed. The test tool 304 also logs the test results and 
progress into log files 302 for further analysis. 

Script oriented test tool 304, typical of current test 
procedures, is external to application program 306. Due to 
this limitation, script oriented test tool 304 can only apply 
external stimuli that are expected, and accepted, as external 
user inputs by the application user program 306. No inter- 
mediate levels of functionality testing below that which is 
made visible to the user interface is possible with this 
external testing approach. Similarly, script oriented test tool 
304 can only detect responses from application program 306 
that are made externally visible by the application program. 
Script oriented test tool 304 is thus limited in its internal 
knowledge of the structure and design of the application 
program 306 and limited to only those stimuli and responses 
that are provided to the external user interface of application 
program 306. For example, a typical response of an appli- 
cation program 306 may be no more than the update of a 
display screen to a user interface. Such a response may be 
detected by script oriented test tool 304 only by capturing 
the screen dump and storing it for subsequent manual review 
by a test operator. In this manner, automation of the testing 
procedures is made more difficult because the detection of 
the application program 306 response is frequently difficult 
to automate. 

Hand generated testing is a variant of the test tool com- 
puting environment depicted in FIG. 3. In a hand generated 
test environment, a test engineer hand generates a desired 
external stimulus rather than receiving a stimulus from a 
pre-defined script file 300, Such hand generated testing 
limits the reproducability and automation of the test proce- 
dures in terms of the generation of external stimuli. The test 
engineer must be certain that all steps which lead up to 
production of a problem are noted so that the problem may 
be reproduced. So called "macro" languages are yet another 
variant of the test procedure depicted in FIG. 3. Macro 
language test tools are similar to the script oriented test tool 
304 but with reduced capabilities for detecting and recording 
the response from application program 306. In essence, 
macro language test tools provide a shorthand notation for 
defining a test script of external stimuli to be applied in 
sequence. 

The well known application program development test 
cycle and associated tools depicted in FIGS. 1 and 3, 
therefore, have several weaknesses as compared to the 
techniques disclosed by the present invention. Specifically, 
script oriented test tool 304 is only capable of testing those 
functions made externally visible by the application program 
306. Similarly, detection of responses generated by appli- 
cation program 306 is limited to those responses made 
visible by the application program. Some such responses 
may be difficult to capture or detect in any automated way 
by script oriented test tool 304. An additional problem with 
the script oriented test tool 304, of FIG. 3, arises in the need 
to port the test tool 304 to each new computing environment 
308. The application program 306 development team nor- 
mally ports the application program 306 to several different 
computing environments 308. In addition, either the quality 
assurance or testing team or the development team must port 
the script oriented test tool 304 and/or the script files 300 to 
the same collection of computing environments 308 inde- 
pendent of the efforts to port the application program 306. 
This imposes an additional work load upon the product 
development and test in addition to the normal load for 
developing the product and identifying problems in the 
application program 306. 
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OVERVIEW— PRESENT INVENTION: 

The testing tools and methods of the present invention 
involve embedding of test operations within the structure of 
the application program during development. By so embed- 
ding test operations within the application program, the 
development team (with the aid and consultation of the test 
team) determines appropriate functions and operations to be 
tested. In addition, the development team determines the 
expected proper response for each identified test operation 
embedded within the application program. The testing tools 
and methods of the present invention further provide a 
standardized, easy to use, graphical, user interface for the 
performance of test operations on the applications program. 
By embedding test operations within the application 
program, complete automation of test procedures can be 
achieved both in provision of stimuli by calling functions, 
and in the capture of test results by verifying the return code 
value of a called test operation function or by direct assertion 
function calls of the application program to detect failures. 
These automated test procedures may be used throughout 
the development portion of the software life cycle by the 
development team or by the test team. This integration of the 
testing of the application program with its development 
eliminates the distinct division of labor typical of prior 
product testing approaches. Knowledge of the program 
structure is no longer required to effectively and robustly test 
the application program. The development team and the test 
team may act in concert as a unified group to assure quality 
in the application program product 

FIG. 2 depicts a software development and test process 
which may utilize the tools and methods of the present 
invention. Unlike the process depicted in FIG. 1, there is no 
vertical dashed line indicating the separate responsibilities 
of the development team and those of the quality assurance 
or testing team. Since there is no longer the need for strict 
division of responsibilities under the methods of the present 
invention, the two teams are referred to under the methods 
of the present invention as the product development group. 
The product development group, by operation of elements 
200 and 202, defines the requirement specifications for the 
application program and develops appropriate program and 
data structures to implement those requirements. In element 
202, the product development group embeds test operation 
object definitions within the program structure of the appli- 
cation program under development. These test operations 
object definitions are derived from object class definitions 
provided by the tools and methods of the present invention. 
By embedding the test operation object definitions within 
the application program (in element 202), the product devel- 
opment group aids in the testing of the application program 
by providing interfaces for testing functionality not other- 
wise visible external to the application program structure. 
This embedding of test operation objects allows automation 
of the testing procedure for generating test stimuli as well as 
automated capture and analysis of the test results. In 
addition, the embedding of the test object definitions permits 
a broader range of test functionality to be provided as 
compared to the external testing approach depicted in FIGS. 
1 and 3. By operation of elements 204 and 206, the product 
development group performs regression testing of the appli- 
cation program by use of the test tools and methods of the 
present invention as embedded within the application pro- 
gram by the test operation object definitions. In elements 
204 and 206, the product development group determines 
whether too many problems are revealed through a verifi- 
cation and regression testing. If too many problems are so 
revealed, the product development group repeats the steps of 
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elements 202-206. The product development group next 
tests the application in elements 208 and 210 to determine if 
too many new problems have arisen in the current version of 
the application program. If the application program is deter- 
mined to be sufficiently problem free by operation of ele- 
ment 210, then the program is released to application users 
as duplicated in element 214. In either case, all newly 
discovered problems are reported for the development group 
to consider as shown in element 212. The reported open 
problems returned to the product development group to 
repeat the program development cycle in hopes of repairing 
the problems. 

The tools and methods of the present invention permit the 
shift in testing paradigm depicted as between FIGS. 1 and 2. 
Whereas under prior approaches all testing, regression 
testing, as well as new testing, was the responsibility of a 
distinct quality assurance and testing team, the tools and 
methods of the present invention permit highly automated 
testing to be performed at any time in the application 
program development lifecycle. 

The tools and methods of the present invention are shown 
in a block diagram of FIG. 4. Application program 404 is 
operable within computing environment 408. Embedded 
within application program 404 are test operation objects 
406 adapted to permit automated testing of functions within 
application program 404. The test operation objects 406 are 
derived (by the application programmer developers) from an 
object class provided in a programming library of the 
present invention. The library is provided in the form of a 
dynamic link library (DLL) to be invoked by application 
program 404. The test operations object library 406 receives 
pre-recorded pseudo-random test suite files or playback files 
400, and logs all testing procedures to playback files and 
results to log files 402. As shown in FIG. 4, the tools and 
methods of the present invention embed the test operations 
406 within the application program 404 at the time of its 
development by the product development group. By embed- 
ding the test operation objects library (DLL) 406 within 
application program for 404, the developers may expose 
more functionality for testing than is otherwise possible in 
prior test methods which exercise only externally visible 
user interface functions. In addition, the results of each test 
operation may be automatically verified by validation of the 
return code value from the invocation of each test operation 
object. An additional benefit of the test tools and methods of 
the present invention is derived from the fact that porting of 
the application program to a particular computing environ- 
ment 408 automatically performs the required porting of the 
test operations embedded within the application program. 
There is no additional effort required to port testing tools to 
each computing environment along with the application 
program itself. 

TEST TOOL OPERATIONS: 

FIG. 5 is a flowchart of the operation of an application 
program which has embedded the test tools and methods of 
the present invention. Element 500 represents the normal 
initialization procedures for the application program. These 
operations include whatever data and code initialization is 
required to properly perform the intended function of the 
application program. Element 502 is next operable to instan- 
tiate all the test operation objects defined by the product 
development group (and derived from the object class 
definition provided by the DLL). The instantiation of the test 
operation objects involves creating the objects and the 
associated interface functions and making them known to 
the test tools which actually perform the testing sequences. 
In addition, as discussed below, element 502 is operable to 
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define groups of related objects for simplifying the user 
interface to the test tool. Instantiating each test operation 
object provides the definition of the objects to be tested by 
the testing tools of the present invention. A table is built 
within the test tools of the present invention in which each 
entry corresponds to one of the defined test operation object. 
As discussed below in further detail, grouping of related 
objects and weighting of the various objects are utilized in 
the automated testing procedures to select appropriate tests 
to be invoked. 

Element 504 determines whether the user has requested 
that testing begin immediately or be deferred until some 
subsequent time in the processing of the applicable program. 
If the testing is to begin immediately, processing continues 
below with element 510. If in the alternative, testing is 
deferred until some subsequent time, processing continues 
with elements 506 and 508 repetitively until it is determined 
by element 508 that the user has requested the commence- 
ment of testing procedures. Until that time, element 506 
represents all normal processing of the application program 
performing its intended function. The determination of 
whether to begin testing immediately or to defer until a later, 
user specified, time is a matter of design choice. One of 
ordinary skill in the art will recognize the potential value in 
deferring the start of testing until a particular state of the 
application program is created (by manual operation of the 
program). Many equivalent design choices may be made by 
those practicing the present invention to implement a desired 
delay in the commencement of a test sequence. 

When testing procedures are begun, elements 510 and 514 
are operable to determine which of three possible modes of 
testing operation are to be performed. The test tools of the 
present invention may be operated in a random auto-test 
mode, a playback auto -test mode, or a manual playback 
creation mode. Element 510 is operable to determine if 
random auto-test mode has been selected. If so, element 512 
is next operable to perform the random auto-test procedures. 
If not, then element 514 is operable to determine if the 
playback auto-test mode has been selected. If so, element 
516 is operable to perform playback auto-test operations. 
Otherwise, element 518 is operable to perform manual 
playback creation procedures. Once begun, elements 512, 
516, and 518 are operable until completion of the testing 
procedures. 

FIG. 6A depicts a flowchart of the detailed operation of 
random auto-test element 512 of FIG. 5. In the random 
auto-test mode, test operation objects are randomly selected 
for invocation to test the application program. The user may 
provide weighting values to increase the frequency of cer- 
tain groups of tests being selected as compared to other 
groups of tests as will be discussed in additional detail 
below. Element 600 is operable to prompt the user for input 
used to adjust the default weighting values applied to each 
test operation object. Based on the default weighting values, 
plus any adjustments entered by the user in operation of 
element 600, element 602 is next operable to generate 
weighted selection tables used for randomly selecting each 
of the test operation objects according to the weighting 
provided by the user. 

A weighting table specifies a range of random number 
values which correspond to each entry in the table. The 
ranges of possible random numbers is equal to the total of 
the weight values for all entries in the corresponding table. 
A weighting table is first generated for the current group 
weight values to determine which group will be selected. 
When selecting a test operation object, a first random 
number is generated in the range of zero through the total of 
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all group weights (minus one). The group whose assigned 
range of numbers encompass the randomly generated num- 
ber is then selected. After the group is randomly selected, a 
weighted selection table is generated for the test operation 
objects according to the current weight values for test 
operation objects in the selected group. A second random 
number is then generated to select from the test operation 
objects defined as members of the selected group. The test 
operation object whose assigned range of numbers encom- 
pass the second randomly generated number is then selected. 
For example, if test objects "A" and "B" are in group "X" 
and test objects "C" and "D" are in group "Y", and the object 
and group weights are as follows: 



Object/Group 



Group Weight 



Object Weight 



X 

y 

A 
B 
C 
D 



60 
30 



20 
10 
100 
150 



Then the group weighted selection table would reflect the 
group weights as follows: 



Group 



Random Number Range 



X 
Y 



0-59 
60-89 



Group X Object 



Random Number Range 



A 
B 



0-19 
20-29 



Group X Object 



Random Number Range 



0-99 
100-249 



10 



15 



20 



25 



30 



The test operations weighted selection table if group X were 
randomly selected would appear as follows: 



35 



40 



The test operations weighted selection table if group Y were 
randomly selected would appear as follows: 



45 



50 



Element 604 is next to operable to prompt the user to enter 
a random seed value. The random selection in the random 
auto -test mode is performed via pseudo-random number 
generation techniques. A sequence of "random" numbers 
generated by such a pseudo-random number generator tech- 
nique may be repeated if the random number generator is 
provided with the same seed value. Such techniques are well 
known to those of ordinary skill in the art and need not be 
discussed further. In order to assure reproducability of any 
problems found, the random number seed value used to 
previously produce the problem may be manually reentered 
by the user so as to reproduce an identical "random" 
sequence of test operations. Element 606 is next operable to 
prompt the user for input parameter values indicating the 
threshold for the number of errors allowed and the total 
count for the number of test operations to be performed. The 
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threshold and count values are used to determine the length 
of time the test procedure will be run and a threshold limit 
at which testing will be halted. If the number of problems 
revealed is below the threshold value testing will proceed, 
otherwise testing is halted when the number of errors 
exceeds the threshold value. Similarly, the total test count 
value is used to determine the number of random test 
operation objects to be invoked in the random auto-test 
mode. 

Elements 608-626 are performed repetitively until testing 
procedures are halted. Element 608 is operable to determine 
whether the specified error threshold has been exceeded. If 
so, testing operations are halted and the test function has 
completed. If not, element 610 is next operable to determine 
whether the total count of test operations has been 
exhausted. If so, testing is completed. If not, processing 
continues by selecting another test operation object invoca- 
tion. 

For each test operation object invocation, elements 
612-626 perform the actual test operation. Element 612 is 
first operable to adjust the weighted selection table for any 
changes in the weighting values. Changes in the weighting 
values are the means by which rules, discussed below in 
additional detail, may alter the testing strategy as the test 
process proceeds. Element 614 is next operable to generate 
a pseudo-random number value for use in the weighted 
selection table to select the next test operation object. The 
pseudo-random number value may be used as an index value 
into the weighted selection tables as described above, or 
otherwise utilized with data structures, well known by those 
of ordinarily skill in the art, in a manner indicative of the 
weighted selections. Element 616 is next operable to record 
the selected test operation object in a playback file. A 
playback file may be utilized in a subsequent test procedure 
to reproduce a problem revealed in random test sequencing. 
The selected test operation object is recorded in the playback 
files before the test is performed to assure that the playback 
file records each test in sequence before the test operation 
execution potentially hangs the application program 
(indicative of a problem in the application program). The 
test procedures and methods of the present invention also 
permit system exception handling to be intercepted by the 
test methods to detect unexpected results in the test opera- 
tion object invocation. This capability of the present inven- 
tion depends upon the computing environment capabilities 
on which the application program (with the test methods 
embedded) is run. 

Element 618 is next operable to invoke the execute 
interface function of the selected test operation object in 
order to execute the test procedures defined by the product 
development group corresponding, to this selected test 
operation object. The execute function is defined by the 
product development group to perform whatever functional 
processing is appropriate to test the desired function corre- 
sponding to the test operation object. Details of an execute 
function are discussed below with respect to exemplary test 
operation objects. The result's of the execution interface 
function is recorded by operation of element 620 in a result 
log file. An execute interface function may indicate its 
success or failure by returning a specific result code as the 
function value or by calling an assertion function in the 
library methods of the present invention to indicate an 
erroneous test result. The result log file may be inspected 
later to determine whether the test sequence failed or 
succeeded, and in addition, may be used to determine at 
which step in the test procedures, as indicated in the play- 
back file, the test sequence failed. Element 622 is operable 
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to determine whether the test execute function result repre- 
sents a successful or failed invocation of the execute inter- 
face function. If the execute interface function invocation 
resulted in an error, element 624 is next operable to incre- 
ment the error counter. If the execute interface function s 
resulted in a successful completion, then the error counter is 
not incremented and operation of element 624 is skipped. 
Element 626 is finally operable to decrement the total test 
count. The test process then continues by looping back to 
elements 608 and 610 which verify that test processing 10 
should continue or should be halted based on the error count 
and the total test count remaining. 

FIG. 7 depicts a flowchart providing additional details of 
the operation of elements 516 of FIG. 5, which performs 
playback auto-test mode processing. Element 700 of FIG. 7 15 
is first operable to prompt the user to enter the name of a 
previously recorded playback file. The playback file con- 
tains information used by the methods of the present inven- 
tion to reproduce a previously recorded test sequence. The 
playback auto-test mode is most typically used to reproduce 20 
a problem previously produced in a sequence of test steps 
(such as the pseudo -random auto-test mode or manual 
playback creation mode discussed above) and recorded in a 
playback file. Test sequence reproducability is key to the 
product development group in locating and repairing pro- 25 
gramming errors (bugs) causing the reported problem. 

Elements 704-710 are next operable repetitively until 
testing procedures have been completed. Element 704 is 
operable to determine whether additional test operation 
objects remain to be processed from the playback file. Each 30 
entry in the playback file represents the invocation of a 
selected test operation object. Element 704 then determines 
whether additional entries are yet to be processed in a 
selected playback file. If no further test operation objects 
remain unprocessed, processing of the playback auto-test 35 
mode is completed and testing halts. If additional entries 
remain in the currently selected playback file, element 706 
is next operable to invoke the execute interface function to 
perform the next selected test operation. 

Element 708 is operable to determine whether the result 40 
of the previous execute interface function indicated success 
or failure. If the execute interface function returned a 
successful completion code, processing continues by loop- 
ing back to element 704 to thereby continue the playback 
auto-test mode by selecting the next test operation from the 45 
playback file. Conversely, if the execute interface function 
invocation indicates an erroneous result, element 710 is next 
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user to select a particular pre-existing playback file (for 
editing) or to create a new playback file. Next, element 802 
prompts the user to identify a test operation object from a 
menu of all defined and instantiated test operation objects. 
Element 804 is next operable to add the selected test 
operation object to the current playback file (alternatively, 
the user may identify a test operation object to be deleted 
from the playback file, to be edited, or to be moved to 
another position in the playback file). Element 806 deter- 
mines whether the user wishes to further edit the playback 
file or whether all changes are done. If further changes are 
desired, execution continues by looping back to element 
802. If no further changes are desired, the method completes 
with processing by element 808 to save the new playback 
file to mass storage associated with the present invention. 
TEST OPERATION OBJECTS: 

Test operation objects are objects derived from a standard 
C++ class definition provided by the library (DLL). The 
object class may be used by the product development group 
to derive specific definitions and instantiations of test opera- 
tion objects. The test tools of the present invention define a 
minimum of four interface function methods for use by the 
testing tools in performing the required test operation. The 
product development group members define their own test 
operation object class which implements at a minimum the 
execute interface methods described above. The other three 
of the four basic interface methods are provided with default 
definitions in the object class. The programmer may use the 
standard object definitions of the three other interface 
methods, or may replace them (override them) as appropri- 
ate for the particular objects test goals. The specific func- 
tionality within each of the four basic interface functions is 
provided by the product development group engineer to 
actually perform the required test operation and return a 
success or failure result. The basic four interface functions 
are: execution, data input, file and memory I/O, and rules. 

The execute interface function must be provided in the 
derived class object. There is no default implementation of 
an execute function because the execute function contains 
all of the code necessary to perform a particular test opera- 
tion. The execute interface function performs only the 
precise function calls required to execute the desired test 
operation function. Any input or output requirements are 
handled through the data input or file and memory I/O 
interface functions. The following is a simple example of an 
execute function used to perform an open file operation. 



DWORD COpenFile::Execute() 
{ 

// Log will write the provided string to the logfile and return FALSE if the 
// operation should skip execution, 
if (Log(m_strOperationName)) 

GetApplicationO->OpenFiIe(m_strFileName); 

return ErrorQ; // Return success or failure. 



operable to record the erroneous result in a results log file. 
Processing then continues by looping back to element 704 to 
complete the playback auto-test mode processing of the 
selected playback file. 

FIG. 8 is a flowchart providing additional detail of the 65 
operation of manual playback creation mode element 518 of 
FIG. 5. Element 800 of FIG. 8 is first operable to prompt the 



This exemplary execute function (the Execute function of 
the COpenFile test operation object) simulates a users 
selection of a file/open menu item exactly as though the user 
has selected the file/open menu item by operation of the 
application program. 

At least two functions may be defined as part of the data 
input interface method. These functions are invoked indi- 
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rectly by the invocation of the execute interface function 
when data input is required. A first function in the data input 
method can automatically generate a desired input value. A 
second function in the data input interface method allows 
manual input of a data value by prompting the test user. The 
following are example functions implementing a typical 
automatic generation, and manual, data input function. 



BOOL COpenFile-GeneratcO 
{ 

CFileList *pFilcList - NULL; 

// Ask the application for a list of files with a certain extension. 
pDileList - GetApplicationQ->FindFilcs( a *.bit"); 
// If there are none, then fail, 
if (pFileList()->EmptyO) 

return FALSE; 
// Pick one of the files randomly from the list. 
nPickedFile = Random(prLleListO->NumberOfHlesQ); 
// Get the name of the picked file and save it. 
m_strFileName = pFileListO->GetFileName (nPickedFile); 
return TRUE; 

} 

BOOL COpenFilenManualO 
{ 

CFileDialog dlgFileDialog; 
BOOL bRet = FALSE; 

// Bring up a dialog box asking the user for a file name. 

if (TOOK = = dlgFileDialog-DoModalO) 

{ 

// [f the user clicks 'OK' save the new file name, 

// otherwise FALSE will be returned and the edit/create 

// operation will be canceled; 

nx_strFileName - dlgFileDialog.GetFileNameQ; 

bRet - TRUE; 

} 

return bRet; 



In general, the Generate function is used by the testing 
tools to create pseudo-random test data input values. Other 
methods besides random generation may be used in a 
Generate function to automatically create appropriate test 
values. For example, sequencing through a variety of test 
values from low to high may be appropriate in some test 
cases. The automatically generated values are also logged in 
the playback file so that any problems revealed by the test 
sequence with specific input values may be easily repro- 
duced. 

The exemplary Manual input function presents exemplary 
code used to display a dialog box in which a test user of the 
application may enter data input values to proceed in the test 
process. As in the Generate exemplary function, any data 
input value received is logged in the playback file so that a 
test sequence may be easily reproduced. 

The file and memory I/O function interface method per- 
mits simulation of input data values retrieved from tempo- 
rary files or memory buffers. Such file or buffered I/O 
requests generated by invocation of the execute function of 



)8,403 Bl 

16 

the test operation object are satisfied by simulating the 
writing and reading of values to and from files or memory 
buffers by the application program. By simulating these 
operations, the values used may be captured and thereby 

s reproduced in later invocations of the playback file (i.e. to 
reproduce a test sequence which resulted in a failure). 

The rules interface method comprises a set of functions 
which determine the propriety of running a particular test 
operation function, or group of test operation functions in 

10 light of the current context of the application program, or the 
context of the test processing. The rules interface method is 
discussed below in additional detail. 
TEST OPERATION OBJECT GROUPS: 
As a convenience to the testing user of an application 

15 program with the test tools embedded according to the 
present invention, functions are provided in the test opera- 
tion object library (DLL) of the present invention to permit 
a development engineer to associate related test operation 
objects into "groups". Function calls in the DLL permit a 

2Q development engineer to create new group definitions. Each 
group becomes associated with a unique group identifier. A 
textual name may be associated with each group for use in 
prompting a test user to enter parameters associated with the 
group. Function calls within the DLL used to define each test 
operation object include a parameter to identify the pre- 

25 defined group with which the test operation object is to be 
associated. 

Each of the groups defined by the development engineer 
may be associated with a weighting value. Weighting values 
are used in conjunction with the pseudo-random auto -test 

30 mode described above to adjust the frequency distribution of 
random calls to the various test operation objects. The 
weighting value is used to determine the relative number of 
calls to test operation objects within one group as distinct 
from the number of calls to test operation objects associated 

35 with another group. In other words, the development engi- 
neer may define one group to be "randomly" selected more 
frequently than is another group. In addition to the group 
weighting, each test operation object within a group may be 
weighted further to determine its relative frequency of 

40 invocation as compared to each other test operation object in 
the same group. 

EMBEDDING TEST OBJECTS: 

The test objects defined by the product development team 
must be defined and instantiated in the initialization code of 

45 the application program as shown in element 502 of FIG. 5. 
The following C++ code sample is exemplary of an 
approach to embedding the test object definition and instan- 
tiation in the application program. In the following exem- 
plary code, the test methods and structures of the present 

50 invention are referred to by the Microsoft trade name 
Test Wizard. It is presumed in the exemplary code below that 
the appropriate header file(s) have been included to provide 
the required object class definitions. 



// Create the test suite object which defines the parameters of the pseudo -random 

// auto-test mode of operation. This object defines the groups of test operations 

// and the relative weights among the groups and operations. 

m_pTcstSuite = new CTestSuite(MAX_GROUP, MAX_OPERATION, "c:\\wiz"); 

// Create the TfestWizard object - the heart of the methods and structure of the 

// present invention. 

m_pTestWizard = new CTestWizard; 

// Define five groups and their respective weights in the test suite object 

// (initially, default values arc used for the group weights) in tbo test suite object. 

// The five groups are: 

// CREATEGROUP 

// FI LEG ROUP 
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-continued 



//MANIPGROUP 
// LAYERGROUP 
// SELECTIONGROUP 

m^pTtstSuitc^SctGroupWcigh^CREATEGROUP, DEFAULT„WEIGHT); 
m^pTestSuitc-^tGroupWeight^FILEGROUP, DEFAULT_ WEIGHT); 
m^pTbstSuite^SetGroupWeightO^ANIPGROUP, DEFAULT_WEIGHT); 
m_plestSuite->SetGroupWeight(LAYERGROUP, DEFAULT_WEIGHT); 
m_pTestSuite->SetGraupWeight(SELECTIONGROUP, DEFAULT_WEIGHT); 
m_pTestWizaid->SetTestSuite(m_pTestSuite, ".sui", ".wiz"); 
m_pTestWizard->SetApplicationName("DemoApp"); 
m_pTestWizard->SetPlaybackExtensioa(".wiz"); 
// Get access to the TestWizard User Interface object 
m_pTestWizard->GetTestWizardUI(&ptestWizUI); 

// Define the pages to be displayed in the UI along with the OPTIONS page 

ptestWizUI->AddPage(newCCreatePage(ptest\VizUI)); 

ptestWizUI->AddPage(newCFUePage(ptestWizUI)) ; 

ptestWizUI->AddPage(new CManipulationPage(ptestWizUl)); 

ptestWizUI->AddFage(newCUyerPage(ptestWizUI)); 

ptestWizUI->AddPage(newCSelection(ptestWi2UI)); 

//Add test operation objects to the FILE group. The test operations are: 

// NEWOP (use the CNewFile test operation object) 

// OPENOP (NotYet Implemented) 

// CLOSEOP (use the CCloseFile test operation object) 

/; SAVEOP (use the CSaveFile test operation object) 

// SAVEASOP (Not Yet Implemented) 

m_p1estSuite->SetOperation(FILEGROUP, NEWOP, new (SKETCHNEWGROUP) 

CNewFile(m_pTestWizard)); 
m_pIestSuite->SetOperation.(FILEGROUP, OPENOP, new 

CNYIOperation(m_p'IestWizard)); 
m_pTestSuite->SetOperation (FILEGROUP, CLOSEOP, new (SKETCHNEWGROUP) 

CCloseFile(m_pTestwizard)); 
m_pTestSuitc->SctOperation(FILEGROUP, SAVEOP, new (SKETCHNEWGROUP) 

CSaveFile(m_pTestWizard)); 
m__pTbstSuitc->SctOperation(FILEGROUP, SAVEASOP, new 

CNY10peration(m_pTestWizard)); 
// Invoke the UI to display the user interface and begin testing. 
m_pTbstWizard->invoke UIQ ; 



The above, exemplary intialization code may be inserted 
in the application program at any desirable position in the 
application. The invocation of the TestWizard User interface 
(to commence testing operations) may be inserted in the 
application at any desirable position including: at startup of 
the application as shown above, following operation of some 
particular function, as an option in the standard user inter- 
face of the application (i.e. in a menu item of the application 
program), or at any other desirable point in the operation of 
the program. 

The SetOperation function calls specify the test operation 
object (third parameter) to be associated with the group and 



of the present invention (and included by appropriate header 
files in the exemplary code above). The CNewFile, 
CCloseFile, and CSaveFile test operation objects are derived 
from the COperation object class. The CNYI Operation 
subclass of the COPERATION class is defined by the 
methods and structures of the present invention to provide a 
default test operation object as a place holder for functions 
not yet implemented by the development group. 

A COPERATION class object may be derived from the 
base class in accord with the following C++ code sample: 



40 



CNewFile: :CNewFUe (CTestWizard *pTestWizard) : COperation (plestWizard) 
{ 

// Body of code to implement a new file operation 

} 



operation specified by the first two parameters. The test The execute interface method function must be imple- 
operation objects are derived from a COPERATION test mented in every derived object. A skeletal exemplar)' 
operation object class defined by the methods and structures execute function might appear as follows: 



DWORD CNewFile: :Executc() 
{ 

CString strMcssagc; 

strMessage.Format("File New Called\n"); 
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-continued 



if (Log (s trMcssagc)) 

AfxGctMainWnd0->ScndMessagc(WM_COMMAND, I D„FILE__NEW) ; 
return Error 0; 

} 



One of ordinary skill in the art will recognize that the 
above exemplary code segments are intended only as 
demonstrative of the structure of actual code used to imple- 
ment and utilize the features of the present invention. Many 
assumptions regarding variable types and scopes are made in 
the above code segments. One of ordinary skill in the art will 
readily recognize the intended meaning of the code sample, 
namely: exemplary pseudo-code suggestive of the intended 
use of the methods and structures of the present invention. 
PSEUDO-RANDOM AUTO-TEST MODE USER INTER- 
FACE: 

FIG. 9 is an exemplary display screen used in the test tools 
and methods of the present invention to permit the test user 
to define the weight values used for each test operation 
object and for each group of test operation objects. The set 
of test operation objects and their groupings are defined by 
the product development group when the objects are embed- 
ded in the source code of the application program (as 
discussed above with respect to FIGS. 2 and 4. Label 900 of 
FIG, 9, discussed in detail below with respect to FIG. 10, 
indicates the hidden "OPTIONS** display screen used to set 
global option values for the pseudo-random auto-test mode 
operation. In FIG. 9, four exemplary groups have been 
defined as indicated by the labels 902-910. Specifically the 
groups are named: "CREATION/DELETION" (label 902), 
"FILE" (label 904), "MANIPULATION" (label 906), 
"LAYER" (label 908), AND "SELECTION" (label 910). 

The display screen of FIG. 9 is an exemplary screen for 
entry of weighting values by the test user to be associated 
with the test group named "CREATION/DELETION." The 
group weight value entered in field 912 sets the group weight 
for this test operation group relative to all other test opera- 
tion groups defined by the product development group. For 
example, if the other four groups (labeled "FILE", 
"MANIPULATION", "LAYER", and "SELECTION") have 
group weights of 75, 25, 100, and 125, respectively, then the 
test operations in this test group ("CREATION/ 
DELETION") will be selected (50/(50+75+25+100+125)) 
times or 13.33% of the pseudo-random selections. 

Within each test group, each test operation may be indi- 
vidually weighted and enabled/disabled. In the 
"CREATION/DELETION" test group of FIG. 9, there are 
six test operations labeled: "CREATE", "DUPLICATE", 
"PASTE", "CUT", "COPY", and "DELETE." These six test 
operations may be individually enabled or disabled by 
checking or clearing the associated check box, 954-964, 
respectively. In addition, each of these six test operation may 
be individually weighted by entering a weight value in the 
associated weight field, 814-924, respectively. The enabled 
test operations within the group will be pseudo-randomly 
selected according to the ratio of their respective weights 
whenever their containing group, "CREATION/ 
DELETION", is selected. In other words, the "CREATE" 
test operation will be selected 50/(50+10+25) times, or 
58.82% of the times when the "CREATION/DELETION" 
group is selected. Since the weighting of the group is 
multiplied by the test operation weight, the "CREATE" test 
operation will be selected 7.84% of the time among all 
defined and enabled test operation objects. 



Button fields 926-932 are used to save and retrieve the 

10 values and settings entered in the above discussed fields 
(912-924 and 954-964) for future reference. Use of buttons 
such as these are well known to those of ordinary skill in the 
art and need not be discussed further. 
Button field 934 begins the pseudo -random auto-test 

15 mode operations as defined by the parameters entered in the 
above discussed fields (912-924 and 954-964). FIGS. 5-8, 
discussed above, present the detailed operation of the 
pseudo-random auto-test mode. 

FIG. 10 is an exemplary display screen used to prompt the 

20 test user of the present invention to enter parameters 
(options) used in the pseudo-random auto-test mode of 
operation. Label 900 of FIG. 10 indicates the semantic use 
of the screen display for entry of "OPTIONS" in the running 
of the pseudo-random auto-test mode. Labels 902-910, as 

25 discussed above with reference to FIG. 9, indicate the names 
of exemplary groups of test operation objects as defined by 
the development. 

Fields on the "OPTIONS" screen of FIG. 10 are used to 
define parameters which control the operation of the pseudo- 

30 random auto-test mode of the present invention. Field 1012 
is used to enter the overall error or failure threshold value. 
This value determines the maximum number of erroneous 
results permitted of all test operation invocations before the 
pseudo-random auto -test mode is terminated. Use of this 

35 overall threshold value is discussed above with respect to 
element 608 of FIG. 6A and elements 622 and 624 of FIG. 
6B. Field 1014 of FIG. 10 is used to enter a single failure 
threshold value for use in the methods of the present 
invention as discussed above with reference to FIGS. 5-8. 

40 The single failure threshold value is used to determine 
whether any particular test operation has exceeded this 
failure threshold. The test tools and methods of the present 
invention account for the number of test operation failures 
associated with the invocation of each test operation object 

45 as well as the total number of test failures. This threshold 
value is used in a manner similar to that of the overall test 
failure threshold to terminate the invocation of a particular 
test operation object in the pseudo-random auto-test mode of 
operation. As shown in FIG. 10, threshold values of zero (or 

50 less) will terminate the pseudo-random auto test mode (field 
1012) or the continued invocation of a particular test (field 
1014) as soon as any failure occurs in the invocation of a test 
operation object. 

Field 1016 is used to enter the random number generator 

55 seed value for the start of the pseudo-random test selection. 
To reproduce a particular "random" test sequence, the user 
may enter the same seed value as used in a previous run of 
that test sequence. Field 1018 is used to enter the total 
number of test operation objects to be invoked for the 

60 pseudo-random test sequence. The pseudo-random auto-test 
mode will terminate operation after this number of test 
operation objects are invoked (regardless of the success or 
failure of the test operation). Field 1020 specifies the direc- 
tory on a mass storage device associated with the computing 

65 environment running the test. Files used in the operation of 
the pseudo-random auto-test mode (playback and result log 
files, etc.) will be stored in this directory for later reference. 
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Check box field 1022 is used to enable the persistent Button fields 1118-1126 are used to save and retrieve the 

storage of single operation threshold values and error counts values and settings entered in the above discussed fields 

from previous invocations of test operations to accumulate (1100-1102) for future reference. Use of buttons such as 

the error counters. When field 1022 is checked, test opera- these are well known to those of ordinary skill in the art and 

tions which previously produced erroneous results exceed- s need not be discussed further, 

ing the designated threshold value are skipped. Otherwise, RULES* 

all test operations are invoked normally. This feature is A , . , „^ . 

& 1 1 ji_ ji li ■ * 4 ^ shown in FIG. 6B, element 612 is operable to adjust 

useful to proceed beyond known problems ma test sequence . . . ' «. . j * * * 

* i >. ui tu- « - * • c the weight selection tables used in pseudo-random auto-test 

to reveal other new problems. This "persistent error in for- - ° , , , , , 

. . , , _ i 4 j f. u ii 4 * mode lor any changed weights. Rules as used herein, are 

matron is retained and accumulated through all test opera- 10 - t . , J . . B . . . ... t . • 

. j ji i i v* * 4 * functions which are invoked in association with the mvo- 

tion invocations and through a plurality of test sequence .. - 4 . ± - , , . iL , 

r- u • l 44 l- l u i * j cation oi the execute interface function wherein the rules 

runs. Field 1024 is a button which, when selected - ,._ , . f . , 

/« v i jm u 4L ii i *l c function modify the current weight value for one or more 

("clicked ) by the user, will clear the persistent log of 4 , ,. \. 4 * j. c . . . . 

x 1* * • •* » 4- c ii *■ • test operation obiects or groups. By modifying the weight 

erroneous results to again permit testing of all operations in , c .. . , . , . / , 

4 4 r *l j j 4 * . i value of a particular test operation object or a particular 

a test sequence of the pseudo-random auto-test mode. is iL \ - r . , « i_ ■ i_ 

c ij • j * „ , , / group, the rules function may alter the frequency with which 

Field 1026 is used to enter a maximum group/operation 4 • * * * i ^ -m. 

• l* i tu- * i j 4 ♦ 4* 11 certain test execute interface functions are invoked. The 

weight value. This maximum value is used to automatically . , . , Al _ , t 4 . .« , r_ ■ 

. ■ • a j 1 p ii ^ * 11 conditions m which the rules function will modify a weight 

generate weighted values for all groups and for all opera- , r i , «, * i_- . ^ 

? i l *4 imo ■ r i j value for a selected test operation obiect or group, may be 

tions within a group when button 1028 is clicked. . , - - - , . it f j • « T- A. i 

Button fields 1030-1036 are used to save and retrieve the 20 ^P^y defined by the code implementing the rules 

values and settings entered in the above discussed fields 6111011011 88 wntten b * the P roduct development group. 

(1012-1028) for future reference. Use of buttons such as Rules > as defined herein, may be used for example, to 

these are well known to those of ordinary skill in the art and reduce ^ frequency of a particular test operation depending 

need not be discussed further. 011 the current state of operation of the application program, 

Button field 1038 begins the pseudo-random auto-test 25 or for example, depending upon the current state of progress 
mode operations as defined by the parameters entered in the m running the test application program. Changing the f re- 
above discussed fields (1012-1028). FIGS. 5-8, discussed *l uenc y of evocation of a particular test operation object or 
above, present the detailed operation of the pseudo-random g rou P ma y delude, for example, reducing its frequency of 
auto-test mode invocation to zero (i.e. "not available"). For example, func- 
PLAYBACK FILE EDITING USER INTERFACE: 30 tions which Perform an edit menu copy or paste function 

FIG. 11 is an exemplary display screen designed to permit ma y be valid onl y when a file is "open" in the context of the 

a test user to edit the contents of a previously recorded application program. Rule functions may be used to assure 

playback file. In the playback file creation mode of that the execute interface functions for the copy or paste test 

operation,atestusereditapreviouslyrecordedplaybackfile operation objects will never be called until after the test 

to customize the test sequence, or create a new test sequence. 35 operation object execute interface function for the files/open 

In the screen display of FIG. 11, a list of available test menu operation is invoked successfully, 

operations 1100 is positioned on the left side of the screen. The rules interface functions for each test operation object 

A list of the test operations currently selected for the test are invoked to adjust the weighted selection table before the 

sequence, the current script 1102, appears on the right side next random selection is made in the flowcharts of FIGS. 6A 

of the screen. A user may highlight ("clickon") a desired test 40 and 6B. 

operation from the list of available test operations, then click What is claimed is: 

on the "ADD" button 1104 to add the highlighted test 1. A software product having an application program 

operation to the current playback 1102. Conversely, the user including: 

may highlight a test operation in the current script 1102 and A 

click the "REMOVE" button 1106 to remove the highlighted 45 Ration P r °g ram ob J ects operational when executed 

test operation from the current script 1102. b y a com P^ng environment to direct the computing 

In addition to the "ADD" and "REMOVE" functions, a environment to perform an application function; 

highlighted test operation in the current playback 1102 may test objects embedded within the application program and 

be skipped by clicking the "DISABLE" button 1108. A operational when executed by the computing environ- 

pause in the test sequence may be added to the current script 50 ment to direct the computing environment to test the 

by highlighting a test operation in the current script 1102 and application program objects; and 

clicking the "BREAK" button 1110. This marks the high- a storage medium operational to store the application 

lighted test operation such that the operation may pause program objects and the test objects, 

before executing. 2. The software product of claim 1 wherein the test objects 

Any test operation in the current playback 1102 may be 55 are derived from a dynamic link library invoked by the 

altered by highlighting the test operation and clicking on the application program. 

"EDIT" button 1112. 3. The software product of claim 2 wherein the test objects 

Test operations in the current list 1102 may be moved up are operational when executed by the computing environ- 

or down in the current list 1102 relative to other test ment to direct the computing environment to randomly and 

operations by highlighting the test operation to be moved 60 automatically test the application program objects, 
and then clicking on the "MOVE UP" button 1114 or 

"MOVE DOWN" button 1116. ***** 
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